Video saliency prediction using enhanced spatiotemporal alignment network

Due to a variety of motions across different frames, it is highly challenging learn an effective spatiotemporal representation for accurate video saliency prediction (VSP). To address this issue, we develop feature alignment network tailored VSP, mainly including two key sub-networks: multi-scale deformable convolutional (MDAN) and bidirectional Long Short-Term Memory (Bi-ConvLSTM) network. The MDAN learns align the features neighboring frames reference one in coarse-to-fine manner, which can well handle various motions. Specifically, owns pyramidal hierarchy structure that first leverages convolution (Dconv) lower-resolution then aggregates aligned higher-resolution features, progressively enhancing from top bottom. output fed into Bi-ConvLSTM further enhancement, captures useful long-time temporal information along forward backward timing directions effectively guide attention orientation shift under complex scene transformation. Finally, enhanced are decoded generate predicted map. proposed model trained end-to-end without any intricate post processing. Extensive evaluations on four VSP benchmark datasets demonstrate method achieves favorable performance against state-of-the-art methods. source codes all results will be released at https://github.com/cj4L/ESAN-VSP.